[HBase]06 Phoenix入门 六

Phoenix简介、Phoenix安装、Phoenix使用

Posted by 李玉坤 on 2017-10-05

Phoenix简介

  • 构建在Apache HBase之上的一个SQL中间层
  • 可以在Apache HBase上执行SQL查询,性能强劲
  • 较完善的查询支持,支持二级索引,查询效率较高
  • OLTP and operational analytics for Apache Hadoop

小知识插入:
OLTP 联机事务处理 比如关系型数据库 Oracle MySQL
OLAP 联机分析处理 比如hive
目前国内OLTP and OLAP : 只有 tidb 数据库 做的最好【半开源】

Phoenix优势

  • Put the SQL back in NoSQL, 程序员熟知SQL语句
  • 具有完整ACID事务功能的标准SQL和JDBC API的强大功能
  • 完全可以和其他Hadoop产品例如Spark, Hive, Pig, Flume以及MapReduce集

Phoenix vs Hive

为什么phoenix快

  • 通过HBase协处理器,在服务端进行操作,从而最大限度的减少客户端和服务器的数据传输
  • 通过定制的过滤器对数据进行处理
  • 使用本地的HBase Api而不是通过MapReduce框架,这样能最大限度的降低启动成本

Phoenix功能特性

  • 二级索引:抛弃之前的只能根据rowkey做索引,通过过滤器删选数据;自由根据索引的列或者表达式形成备用的行键更方便的进行数据的查找
  • 多租户:通过多租户表和租户专用的链接使用户只能访问自己的数据
  • 用户定义函数:自己实现DF;像select一样使用
  • 行时间戳列:可以将hbase每行的时间戳映射成Phoenix的一个列
  • 分页查询:标准的sql分页语法
  • 视图:标准sql语法(Phoenix可以使用多个虚拟表共享底层的物理表)

Phoenix安装

  • 下载对应版本的安装包
    http://phoenix.apache.org/download.html
  • 配置与现有HBase集群集成重新
    解压后如下操作
    如果是分布式则应拷贝到每个reginserver下
    1
    2
    3
    [hadoop@hadoop lib]$ cp phoenix-core-4.14.0-cdh5.14.2.jar ../../hbase/lib
    [hadoop@hadoop lib]$ cd ..
    [hadoop@hadoop phoenix]$ cp phoenix-4.14.0-cdh5.14.2-server.jar ../hbase/lib/
  • 进入/home/hadoop/app/hbase/conf
    [hadoop@hadoop conf]$ vim hbase-site.xml
    添加:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17

    <property>
    <name>hbase.table.sanity.checks</name>
    <value>false</value>
    </property>
    <property>
    <name>hbase.regionserver.wal.codec</name>
    <value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value>
    </property>
    <property>
    <name>phoenix.schema.isNamespaceMappingEnabled</name>
    <value>true</value>
    </property>
    <property>
    <name>phoenix.schema.mapSystemTablesToNamespace</name>
    <value>true</value>
    </property>
  • 进入phoenix的bin文件夹
    [hadoop@hadoop bin]$ rm -rf hbase-site.xml
    [hadoop@hadoop conf]$ ln -s /home/hadoop/app/hbase/conf/hbase-site.xml /home/hadoop/app/phoenix/bin/hbase-site.xml
    注意点:
    假如 hdfs ha怎么办?
    cdh: /etc/hdfs/conf/core-site.xml 做软连接【phoenix的bin文件夹】
    /etc/hdfs/conf/hdfs-site.xml 做软连接【phoenix的bin文件夹】
  • 启动HBase环境,并测试环境是否可以正常使用
    注意python版本 2.7
    /xxx/xxx/python2.7 ./sqline.py hadoop:2181

前置条件启动HDFS、zookeeper
启动hbase
最后启动phoenix(bin/./sqlline.py)【./sqline.py hadoop:2181】

Phoenix实战:shell命令操作Phoenix

语法:
http://phoenix.apache.org/language/index.html#select
http://phoenix.apache.org/joins.html
http://phoenix.apache.org/subqueries.html

数据类型
MySQL phoenix
INT INTEGER
BIGINT
FLOAT
DOUBLE
DECIMAL
VARCHAR
DATETIME TIMESTAMP
TIMESTAMP TIMESTAMP
CHAR VARCHAR

【注意】

  • CHAR/VARCHAR 长度50=》phoenix中VARCHAR 翻倍 100,防止上游数据结构改变大小。
  • phoenix 支持字段add语法,不支持modify语法(更新字段) 只能drop 再add
  • phoenix库名以及表名不区分,统一转换成大写

进入shell

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[hadoop@hadoop bin]$ ./sqlline.py hadoop:2181
Setting property: [incremental, false]
Setting property: [isolation, TRANSACTION_READ_COMMITTED]
issuing: !connect jdbc:phoenix:hadoop:2181 none none org.apache.phoenix.jdbc.PhoenixDriver
Connecting to jdbc:phoenix:hadoop:2181
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/app/phoenix-4.14.0-cdh5.14.2/phoenix-4.14.0-cdh5.14.2-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/hadoop/app/hadoop-2.6.0-cdh5.14.2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/03/31 20:58:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Connected to: Phoenix (version 4.14)
Driver: PhoenixEmbeddedDriver (version 4.14)
Autocommit status: true
Transaction isolation: TRANSACTION_READ_COMMITTED
Building list of tables and columns for tab-completion (set fastconnect to true to skip)...
133/133 (100%) Done
Done
sqlline version 1.2.0
0: jdbc:phoenix:hadoop:2181>

创建表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
0: jdbc:phoenix:hadoop:2181> !tables
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+----------+
| TABLE_CAT | TABLE_SCHEM | TABLE_NAME | TABLE_TYPE | REMARKS | TYPE_NAME | SELF_REFERENCING_COL_NAME | REF_GENERATION | INDEX_ST |
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+----------+
| | SYSTEM | CATALOG | SYSTEM TABLE | | | | | |
| | SYSTEM | FUNCTION | SYSTEM TABLE | | | | | |
| | SYSTEM | LOG | SYSTEM TABLE | | | | | |
| | SYSTEM | SEQUENCE | SYSTEM TABLE | | | | | |
| | SYSTEM | STATS | SYSTEM TABLE | | | | | |
+------------+--------------+-------------+---------------+----------+------------+----------------------------+-----------------+----------+
0: jdbc:phoenix:hadoop:2181> CREATE schema my_schema;
No rows affected (0.032 seconds)
0: jdbc:phoenix:hadoop:2181> CREATE TABLE my_schema.my_table ( id BIGINT not null primary key, date Date);
No rows affected (1.256 seconds)
0: jdbc:phoenix:hadoop:2181> select * from my_schema.my_table;
+-----+-------+
| ID | DATE |
+-----+-------+
+-----+-------+
No rows selected (0.063 seconds)
0: jdbc:phoenix:hadoop:2181>

插入数据phoenix插入是upsert与普通sql不同

1
2
3
4
5
6
7
8
9
0: jdbc:phoenix:hadoop:2181> upsert into my_schema.my_table(id, date) values (1,'2020-03-31');
1 row affected (0.024 seconds)
0: jdbc:phoenix:hadoop:2181> select * from my_schema.my_table;
+-----+--------------------------+
| ID | DATE |
+-----+--------------------------+
| 1 | 2020-03-31 00:00:00.000 |
+-----+--------------------------+
1 row selected (0.035 seconds)

插入列

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
0: jdbc:phoenix:> alter table my_schema.my_table add sex varchar(10);
No rows affected (6.614 seconds)
0: jdbc:phoenix:> select * from my_schema.my_table;
+-----+--------------------------+------+
| ID | DATE | SEX |
+-----+--------------------------+------+
| 1 | 2020-03-31 00:00:00.000 | |
+-----+--------------------------+------+
1 row selected (0.042 seconds)
0: jdbc:phoenix:> upsert into my_schema.my_table(id, date,sex) values (1,'2020-03-31','男');
1 row affected (0.063 seconds)
0: jdbc:phoenix:> select * from my_schema.my_table;
+-----+--------------------------+------+
| ID | DATE | SEX |
+-----+--------------------------+------+
| 1 | 2020-03-31 00:00:00.000 | 男 |
+-----+--------------------------+------+
1 row selected (0.046 seconds)
0: jdbc:phoenix:>

复杂sql语句

1
2
3
4
5
6
7
0: jdbc:phoenix:> select sex,count(sex) as num from my_schema.my_table group by sex;
+------+------+
| SEX | NUM |
+------+------+
| 男 | 1 |
+------+------+
1 row selected (0.079 seconds)
1
2
3
4
5
6
7
8
9
10
11
12

# Phoenix实战:java jdbc操作Phoenix
引入依赖

```xml
<dependencies>
<dependency>
<groupId>org.apache.phoenix</groupId>
<artifactId>phoenix-core</artifactId>
<version>4.13.1-HBase-1.2</version>
</dependency>
</dependencies>
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
package com.kun.bigdata.phoenix.test;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;

public class PhoenixTest {

public static void main(String[] args) throws Exception {
Class.forName("org.apache.phoenix.jdbc.PhoenixDriver");
Connection connection = DriverManager.getConnection("jdbc:phoenix:localhost:2181");

PreparedStatement statement = connection.prepareStatement("select * from PERSON");

ResultSet resultSet = statement.executeQuery();

while (resultSet.next()) {
System.out.println(resultSet.getString("NAME"));
}

statement.close();
connection.close();
}
}

通过mybatis操作Phoenix

引入依赖

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-logging</artifactId>
</exclusion>
</exclusions>
<version>1.4.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<exclusions>
<exclusion>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
</exclusion>
</exclusions>
<version>1.4.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jetty</artifactId>
<version>1.4.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<version>1.4.2.RELEASE</version>
</dependency>
<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
</exclusions>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis</artifactId>
<version>3.4.2</version>
</dependency>
<dependency>
<groupId>org.mybatis</groupId>
<artifactId>mybatis-spring</artifactId>
<version>1.3.1</version>
</dependency>
<dependency>
<groupId>org.mybatis.spring.boot</groupId>
<artifactId>mybatis-spring-boot-starter</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>org.apache.phoenix</groupId>
<artifactId>phoenix-core</artifactId>
<version>4.13.1-HBase-1.2</version>
</dependency>
</dependencies>

添加配置文件
applilcation.properties

1
2
3
4
5
6
7
8
9
mybatis.config-location=mybatis-config.xml
datasource.jdbcUrl=jdbc:phoenix:localhost:2181
datasource.driverClassName=org.apache.phoenix.jdbc.PhoenixDriver
datasource.maxPoolSize=20
datasource.minIdle=2
datasource.validationTimeout=300000
datasource.idleTimeout=600000
datasource.connectionTestQuery=select 1+1
mybatis.mapperLocations=

log4j.properties

1
2
3
4
5
log4j.rootLogger=INFO
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

mybatis-config.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE configuration
PUBLIC "-//mybatis.org//DTD Config 3.0//EN"
"http://mybatis.org/dtd/mybatis-3-config.dtd">
<configuration>
<settings>
<setting name="cacheEnabled" value="true"/>
<setting name="lazyLoadingEnabled" value="true"/>
<setting name="multipleResultSetsEnabled" value="true"/>
<setting name="useColumnLabel" value="true"/>
<setting name="useGeneratedKeys" value="false"/>
<setting name="autoMappingBehavior" value="PARTIAL"/>
<setting name="autoMappingUnknownColumnBehavior" value="WARNING"/>
<setting name="defaultExecutorType" value="SIMPLE"/>
<setting name="defaultStatementTimeout" value="25"/>
<setting name="defaultFetchSize" value="100"/>
<setting name="safeRowBoundsEnabled" value="false"/>
<setting name="mapUnderscoreToCamelCase" value="false"/>
<setting name="localCacheScope" value="SESSION"/>
<setting name="jdbcTypeForNull" value="OTHER"/>
<setting name="lazyLoadTriggerMethods" value="equals,clone,hashCode,toString"/>
</settings>
</configuration>

UserInfo实体类

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
package com.kun.bigdata.phoenix.mybatis.test;


public class UserInfo {

private int id;
private String name;

public int getId() {
return id;
}

public void setId(int id) {
this.id = id;
}

public String getName() {
return name;
}

public void setName(String name) {
this.name = name;
}
}

HikariDataSourceFactory 相当于配置了一个数据库连接池

1
2
3
4
5
6
7
8
9
10
11
12
13
package com.kun.bigdata.phoenix.mybatis.test.mybatis;

import org.apache.ibatis.datasource.unpooled.UnpooledDataSourceFactory;

import com.zaxxer.hikari.HikariDataSource;


public class HikariDataSourceFactory extends UnpooledDataSourceFactory {

public HikariDataSourceFactory() {
this.dataSource = new HikariDataSource();
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
package com.kun.bigdata.phoenix.mybatis.test.mybatis;

import java.io.IOException;
import java.io.InputStream;
import java.util.Properties;
import java.util.Set;

import javax.sql.DataSource;

import org.apache.ibatis.session.SqlSessionFactory;
import org.apache.ibatis.session.SqlSessionFactoryBuilder;
import org.mybatis.spring.SqlSessionFactoryBean;
import org.mybatis.spring.annotation.MapperScan;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;
import org.springframework.core.io.DefaultResourceLoader;
import org.springframework.core.io.ResourceLoader;


@Configuration
@MapperScan(basePackages = PhoenixDataSourceConfig.PACKAGE,
sqlSessionFactoryRef = "PhoenixSqlSessionFactory")
public class PhoenixDataSourceConfig {

static final String PACKAGE = "com.kun.bigdata.phoenix.**";

@Bean(name = "PhoenixDataSource")
@Primary
public DataSource phoenixDataSource() throws IOException {
ResourceLoader loader = new DefaultResourceLoader();
InputStream inputStream = loader.getResource("classpath:application.properties")
.getInputStream();
Properties properties = new Properties();
properties.load(inputStream);
Set<Object> keys = properties.keySet();
Properties dsProperties = new Properties();
for (Object key : keys) {
if (key.toString().startsWith("datasource")) {
dsProperties.put(key.toString().replace("datasource.", ""), properties.get(key));
}
}
HikariDataSourceFactory factory = new HikariDataSourceFactory();
factory.setProperties(dsProperties);
inputStream.close();
return factory.getDataSource();
}

@Bean(name = "PhoenixSqlSessionFactory")
@Primary
public SqlSessionFactory phoenixSqlSessionFactory(
@Qualifier("PhoenixDataSource") DataSource dataSource) throws Exception {
SqlSessionFactoryBean factoryBean = new SqlSessionFactoryBean();
factoryBean.setDataSource(dataSource);
ResourceLoader loader = new DefaultResourceLoader();
String resource = "classpath:mybatis-config.xml";
factoryBean.setConfigLocation(loader.getResource(resource));
factoryBean.setSqlSessionFactoryBuilder(new SqlSessionFactoryBuilder());
return factoryBean.getObject();
}
}

dao层

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
package com.kun.bigdata.phoenix.mybatis.test.dao;

import java.util.List;

import org.apache.ibatis.annotations.Delete;
import org.apache.ibatis.annotations.Insert;
import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param;
import org.apache.ibatis.annotations.ResultMap;
import org.apache.ibatis.annotations.Select;

import com.kun.bigdata.phoenix.mybatis.test.UserInfo;


@Mapper
public interface UserInfoMapper {

@Insert("upsert into USER_INFO (ID,NAME) VALUES (#{user.id},#{user.name})")
public void addUser(@Param("user") UserInfo userInfo);

@Delete("delete from USER_INFO WHERE ID=#{userId}")
public void deleteUser(@Param("userId") int userId);

@Select("select * from USER_INFO WHERE ID=#{userId}")
@ResultMap("userResultMap")
public UserInfo getUserById(@Param("userId") int userId);

@Select("select * from USER_INFO WHERE NAME=#{userName}")
@ResultMap("userResultMap")
public UserInfo getUserByName(@Param("userName") String userName);

@Select("select * from USER_INFO")
@ResultMap("userResultMap")
public List<UserInfo> getUsers();
}

测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
package com.kun.bigdata.phoenix.mybatis.test;

import java.util.List;

import org.junit.Test;
import org.junit.runner.RunWith;
import org.mybatis.spring.annotation.MapperScan;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.ComponentScan;
import org.springframework.context.annotation.Import;
import org.springframework.context.annotation.PropertySource;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import com.kun.bigdata.phoenix.mybatis.test.dao.UserInfoMapper;
import com.kun.bigdata.phoenix.mybatis.test.mybatis.PhoenixDataSourceConfig;


@RunWith(SpringJUnit4ClassRunner.class)
@Import(PhoenixDataSourceConfig.class)
@PropertySource("classpath:application.properties")
@ComponentScan("com.kun.bigdata.**")
@MapperScan("com.kun.bigdata.**")
public class BaseTest {

@Autowired
UserInfoMapper userInfoMapper;

@Test
public void addUser() {
UserInfo userInfo = new UserInfo();
userInfo.setId(2);
userInfo.setName("Jerry");
userInfoMapper.addUser(userInfo);
}

@Test
public void getUserById() {
UserInfo userInfo = userInfoMapper.getUserById(1);
System.out.println(String.format("ID=%s;NAME=%s", userInfo.getId(), userInfo.getName()));
}

@Test
public void getUserByName() {
UserInfo userInfo = userInfoMapper.getUserByName("Jerry");
System.out.println(String.format("ID=%s;NAME=%s", userInfo.getId(), userInfo.getName()));
}

@Test
public void deleteUser() {
userInfoMapper.deleteUser(1);

List<UserInfo> userInfos = userInfoMapper.getUsers();
for (UserInfo userInfo : userInfos) {
System.out.println(String.format("ID=%s;NAME=%s", userInfo.getId(), userInfo.getName()));
}
}
}